Two Experiments on Subjective Evaluation of Code Evolvability

Licentiate thesis submitted for official examination for the degree of Licentiate in Technology. Software evolvability – the ease of further developing software – is an important quality attribute greatly dictating the future potential of any software system. Recent trends such as agile software development and extreme programming have highlighted refactoring – modifying the internal structure of software without affecting its observable behaviour – as a key factor for ensuring software evolvability. To help software developers in making a refactoring decision, descriptions of bad code, so called code smells have been proposed. Although humans play an important role in making refactoring decisions, most of the work around refactoring has focused on tools and metrics, instead of empirical work with humans. Therefore, we studied refactoring decisions and the evaluation of the existence of code smells at the source code method level. Two experiments were made with a different sample of students in each. The participants evaluated the existence of certain code smells for 10 methods, stated whether each method should be refactored or not, and provided rationales for their refactoring decisions. We studied the interrater agreement, i.e. the extent to which evaluators agree. Furthermore, we studied the explaining factors of the evaluations with three approaches. First, with regression analysis we studied how the measures of the evaluated code and the background of the evaluators affected the evaluations. Second, we analyzed the information of the refactoring decision rationales. Third, we used the information discovered in the rationales to explain the refactoring decision with regression analysis. The results showed high interrater agreement for the simple code smells, and low agreement for the more complex smell and the refactoring decision. The code metrics explained over 70% of the variation regarding the simple code smell evaluations, but only about 30% of the refactoring decision. Surprisingly, the demographics were not useful predictors neither for evaluating the code smells nor the refactoring decision. The best predictors of the refactoring decision were the rationales of the refactoring decision, i.e. the qualitative data, explaining over 70% of the variation, and the evaluations of the code smells explaining more than 60% of the variation. The analysis of qualitative data showed that the method under study affects the contents of the rationales. In other words with different methods we would have obtained different rationales. The qualitative data indicates that automatic detection of some code problems would require new code metrics, and some …